Mixed reality space sharing system, sharing information management server, mixed reality terminal, mixed reality space sharing method, and sharing information management program

ABSTRACT

A mixed reality space sharing system is provided with which 3D information regarding a target real space can be shared between a plurality of MR terminals while keeping the information updated, without spending a long processing time. A shared information management server includes: an information management unit for creating a room by associating, with each other, 3D information and anchor information that are transmitted from a first terminal, which is at least one of a plurality of mixed reality terminals, and managing the information; and a sharing unit for transmitting the 3D information and the anchor information to a second terminal different from the first terminal. The second terminal downloads the 3D information and the anchor information transmitted from the shared information management server, and creates a mixed reality space using the downloaded information.

TECHNICAL FIELD

An aspect of this invention relates to a mixed reality space sharingsystem, a shared information management server, a mixed realityterminal, a mixed reality space sharing method, and a shared informationmanagement program.

BACKGROUND ART

In recent years, with the evolution of augmented reality (AR) andvirtual reality (VR), mixed reality (MR), which integrates them, hasbeen attracting attention. MR refers to a technology that constructs aspace (mixed reality space) where a virtual space and a real space aremixed by projecting virtual information onto the real space. The marketsize of MR is increasing year by year. A typical device for realizing MRis Microsoft's HoloLens (e.g., see NPL 1). HoloLens is equipped with asensor for acquiring a real space as three-dimensional (3D) information.HoloLens mixes a virtual space with the real space by mapping virtualinformation onto a 3D model of the real space acquired by this sensor.HoloLens allows users to experience a mixed reality space by projectingthe thus-constructed mixed reality space onto a transmission holographiclens. MR terminals typified by HoloLens are also gaining attention asnext-generation mobile terminals to replace smartphones (e.g., see NPLs2, 3, and 4). It is considered that, in the future, MR terminals will beused not only in the above examples, but also in more general mannersand in various places, both indoors and outdoors.

Examples of MR applications include those that can be completed by anindividual, such as “projecting a 3D model in a chamber and simulatinglayouts (e.g., see NPL 5). Other examples include “in the constructionand manufacturing industries, presenting instructions from an operatoras virtual information in a site worker's field of view (e.g., see NPL6)” and “in the real estate industry, presenting the figure of afinished building in a planned building construction site as a 3D objectin a customer's field of view (e.g., see NPL 7). Thus, MR is currentlyattracting attention as a technology for sharing virtual information ina real space for collaborative work and decision making between aplurality of people.

Against this background, Microsoft is advocating “collaborativecomputing”. Collaborative computing is a computing model in whichinformation that is currently viewed by each individual via a PC isprojected into a real space and shared by a plurality of people, whointeract with the information as if they were touching a real object. Ifthis computing model becomes more prevalent, the place where informationis viewed and shared will not be a display of a personal computer (PC)or a smartphone, but the real space itself. Thus, the technology forsharing a mixed reality space between a plurality of people in MR isexpected to become increasingly important in the future.

Microsoft provides a space sharing service (hereinafter referred to as“sharing service”) (e.g., see NPL 8) to enable mixed reality spaces tobe easily shared. In this sharing service, information (anchorinformation) for anchoring a virtual object in a mixed reality space isshared between MR terminals via a server in order to share the mixedreality space between different MR terminals. Each MR terminal thatreceives the anchor information displays the virtual object at arelative position with respect to the coordinates calculated based onthe anchor information. Each MR terminal can thus display the virtualobject at the same position in a real space, which is a specific areasuch as a chamber, so that a plurality of users can view the same mixedreality space. Information regarding the shared mixed reality space ismanaged as a room in a server program, and the anchor information isalso stored in this room. In the program, the sharing of the mixedreality space is realized by transmitting the anchor information to MRterminals that are associated with the same room.

CITATION LIST Non Patent Literature

-   -   [NPL 1] “Microsoft HoloLens: Mixed Reality Technology to Support        Business” [Online], [Retrieved on Jun. 24, 2019], Internet <URL:        https://www.microsoft.com/ja-jp/HoloLens>    -   [NPL 2] Taleb, Tarik, et al. “PERMIT: Network slicing for        personalized 5G mobile telecommunications,” IEEE Communications        Magazine, 55(5), pp. 88-93, 2017.    -   [NPL 3] Debandi, Federico, et al. “Enhancing cultural tourism by        a mixed reality application for outdoor navigation and        information browsing using immersive devices,” IOP Conference        Series: Materials Science and Engineering, 364(1), pp. 12-48,        2018.    -   [NPL 4] “How Microsoft Jumped on Apple's Laptop Stumbles—Bloo        mberg” [Online], [Retrieved on Jun. 24, 2019], Internet <URL:        https://www.bloomberg.com/news/features/2017-05-03/microsoft-g        ets-hardware-religion>    -   [NPL 5] “Layout Microsoft Dynamics 365” [Online], [Retrieved on        Jun. 24, 2019], Internet <URL:        https://dynamics.microsoft.com/ja-jp/mixed-reality/layout/>    -   [NPL 6] “Remote Aassist 1 Microsoft Dynamics 365” [Online],        [Retrieved on Jun. 24, 2019], Internet <URL:        https://dynamics.micro        soft.com/ja-jp/mixed-reality/remote-assist/>    -   [NPL 7] “Japan's First! Employ Microsoft Holorens for        Condominium Sales” [Online], [Retrieved on Jun. 24, 2019],        Internet <URL: https://www.nextscape.net/news/2017/05/20170523>    -   [NPL 8] “GitHub—microsoft/MixedRealityToolkit at c562ff9582cd        10ea0448fd846f7b2bb261e8f551” [Online], [Retrieved on Jun. 24,        2019], Internet <URL:        https://github.com/Microsoft/MixedRealityToolkit/tree/c562ff95        82cd10ea0448fd846f7b2bb261e8f551>

SUMMARY OF THE INVENTION Technical Problem

As mentioned above, anchor information for anchoring 3D informationregarding a real space and a virtual object in a mixed reality space ismanaged in association with a virtual room for each specific area(chamber etc.) by a server. As a result of the anchor information beingshared between MR terminals via the server, the same mixed reality spacecan be shared between a plurality of MR terminals. In this case, toactually share the mixed reality space, it is necessary for users tosense every corner of a targe real space in advance for respective MRterminals so that the MR terminals recognize 3D information regardingthe real space. For this reason, the larger the target real space is,the longer it takes to preprocess information.

This invention has been made in view of the foregoing circumstances, andan object of the invention is to provide a mixed reality space sharingsystem, a shared information management server, a mixed realityterminal, a mixed reality space sharing method, and a shared informationmanagement program that enable 3D information regarding a target realspace to be shared between a plurality of MR terminals without spendinga long processing time.

Means for Solving the Problem

To solve the above problem, a first aspect of this invention is a mixedreality space sharing system including: a plurality of mixed realityterminals each of which creates a mixed reality space in which virtualinformation is mapped onto 3D information regarding a real space basedon the 3D information and anchor information for anchoring the virtualinformation, and visualizing the mixed reality space; and a sharedinformation management server to which the plurality of mixed realityterminals are connected, wherein a first terminal, which is at least oneof the plurality of mixed reality terminals, includes: a 3D sensor forsensing the real space and acquiring the 3D information; and an uploadunit for transmitting the 3D information acquired by the 3D sensor andthe anchor information to the shared information management server, theshared information management server includes: an information managementunit for creating a room by associating the 3D information and theanchor information transmitted from the first terminal with each other,and managing the information; and a sharing unit for transmitting the 3Dinformation and the anchor information to a second terminal differentfrom the first terminal, of the plurality of mixed reality terminals,and the second terminal includes a download unit for downloading the 3Dinformation and the anchor information that are transmitted from theshared information management server, and creates the mixed realityspace using the downloaded 3D information and anchor information.

A second aspect of this invention is a shared information managementserver including: an information management unit for creating a room byassociating 3D information regarding a real space and anchor informationfor anchoring virtual information with each other, and managing theinformation, the 3D information and the anchor information being used bya plurality of mixed reality terminals each of which creates a mixedreality space in which virtual information is mapped onto the 3Dinformation based on the 3D information and the anchor information, andvisualize the mixed reality space; and a sharing unit for transmittingthe 3D information and the anchor information to a mixed realityterminal that has made a request, of the plurality of mixed realityterminals, in accordance with the request.

A third aspect of this invention is a plurality of mixed realityterminals each of which create a mixed reality space in which virtualinformation is mapped onto 3D information regarding a real space basedon the 3D information and anchor information for anchoring the virtualinformation, and visualizes the mixed reality space, each of theplurality of mixed reality terminals including: a 3D sensor for sensingthe real space and acquiring the 3D information; and an upload unit fortransmitting the 3D information acquired by the 3D sensor and the anchorinformation to a shared information management server; and an updatedinformation upload unit for periodically comparing the 3D informationacquired by the 3D sensor with immediately previous information, andtransmitting, as updated information, 3D information of only a portionwith a significant change to the shared information management server.

A fourth aspect of this invention is a mixed reality space sharingmethod to be used in a mixed reality space sharing system including: aplurality of mixed reality terminals each of which creates a mixedreality space in which virtual information is mapped onto 3D informationregarding a real space based on the 3D information and anchorinformation for anchoring the virtual information, and visualizing themixed reality space; and a shared information management server to whichthe plurality of mixed reality terminals are connected, the methodincluding: uploading the 3D information acquired by sensing the realspace and the anchor information to the shared information managementserver from a first terminal, which is at least one of the plurality ofmixed reality terminals; creating a room by associating the 3Dinformation and the anchor information uploaded from the first terminalwith each other, and managing the information, in the shared informationmanagement server; transmitting the managed 3D information and anchorinformation to a second terminal different from the first terminal, ofthe plurality of mixed reality terminals, in accordance with a requestfrom the second terminal; and downloading the 3D information and theanchor information that are transmitted from the shared informationmanagement server, and creating the mixed reality space using thedownloaded 3D information and anchor information, in the secondterminal.

A fifth aspect of this invention is a shared information managementprogram to be executed by a plurality of mixed reality terminals each ofwhich creates a mixed reality space in which virtual information ismapped onto 3D information regarding a real space based on the 3Dinformation and anchor information for anchoring the virtualinformation, and visualizes the mixed reality space, and a processor ofa shared information management server to which the plurality of mixedreality terminals are connected, the program causing the processor tofunction as: an information management unit for creating a room byassociating the 3D information and the anchor information used by theplurality of mixed reality terminals with each other, and managing theinformation; and a sharing unit for transmitting the 3D information andthe anchor information to a mixed reality terminal that has made arequest, of the plurality of mixed reality terminals, in accordance withthe request.

Effects of the Invention

According to each aspect of this invention, a mixed reality spacesharing system, a shared information management server, a mixed realityterminal, a mixed reality space sharing method, and a shared informationmanagement program can be provided that enable 3D information regardinga target real space to be shared between a plurality of MR terminalswithout spending a long processing time, by managing the 3D informationregarding a real space corresponding to a room by the shared informationmanagement server.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram that shows an overall configuration of a mixedreality space sharing system according to one embodiment of thisinvention.

FIG. 2 is a block diagram that shows a hardware configuration of an MRterminal serving as a mixed reality terminal according to one embodimentof this invention.

FIG. 3 is a block diagram that shows a software configuration of the MRterminal in FIG. 2.

FIG. 4 is a block diagram that shows a configuration of an initialsetting unit in FIG. 3.

FIG. 5 is a block diagram that shows a hardware configuration of asharing server serving as a shared information management serveraccording to one embodiment of this invention.

FIG. 6 is a block diagram that shows a software configuration of thesharing server in FIG. 5.

FIG. 7A is a diagram that shows first parts of flowcharts showing anexample operation in processing related to sharing of a mixed realityspace between a plurality of MR terminals performed by processing unitsof the MR terminal and the sharing server in FIG. 1.

FIG. 7B is a diagram that shows second parts of the flowcharts showingthe example operation in processing related to sharing of a mixedreality space between a plurality of MR terminals performed by theprocessing units of the MR terminal and the sharing server in FIG. 1.

FIG. 7C is a diagram that shows third parts of the flowcharts showingthe example operation in processing related to sharing of a mixedreality space between a plurality of MR terminals performed by theprocessing units of the MR terminal and the sharing server in FIG. 1.

FIG. 8 is a schematic diagram for illustrating processing for sharing 3Dinformation regarding a real space between two MR terminals.

FIG. 9 is a diagram showing an example of shared information managed bythe shared information management server when 3D information regarding areal space is shared between two MR terminals.

FIG. 10 is a schematic diagram for illustrating processing for sharing3D information regarding a real space when a third MR terminals joins.

FIG. 11 is a diagram that shows an example of shared information managedby the shared information management server when 3D informationregarding a real space is shared between three MR terminals.

FIG. 12 is a diagram that shows a box to be sensed and 3D informationregarding the box.

FIG. 13 is a schematic diagram for illustrating uploading of 3Dinformation regarding a real space.

FIG. 14 is a diagram for illustrating an example of 3D informationchange detection performed by a 3D information change detection unit inFIG. 3.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment related to this invention will be describedwith reference to the drawings.

(Configuration)

(1) System

FIG. 1 is a diagram that shows an overall configuration of a mixedreality space sharing system according to an embodiment of thisinvention. This mixed reality space sharing system is a server-clientsystem in which a plurality of clients are connected to a server. Themixed reality space sharing system includes a plurality of MR terminals10#1, 10#2, . . . , 10#n (n is an integer) (hereinafter also referred tocollectively as “MR terminals 10”), and a sharing server 20. The MRterminals 10 can communicate with the sharing server 20 via acommunication network NW as necessary.

Each MR terminal 10 is a mixed reality terminal according to oneembodiment of this invention that acquires 3D information regarding areal space, and creates and visualizes a mixed reality space in whichvirtual information is mapped onto 3D information based on anchorinformation for anchoring the virtual information.

The sharing server 20 is a shared information management serveraccording to one embodiment of this invention that is constituted by,for example, a server computer, a personal computer in which a serverprogram operates, or the like, creates a room by associating 3Dinformation and anchor information transmitted from the MR terminals 10with each other, and manages shared information regarding the room.

The communication network NW is constituted by, for example, an IP(Internet Protocol) network, which is typified by the Internet, and aplurality of access networks for accessing this IP network. Not onlywired access networks that use optical fiber, but also wireless accessnetworks such as a mobile phone network that operates under the standardsuch as 3G, 4G, or 5G, and a wired LAN (Local Area Network) are used asthe access networks. It is desirable that wireless access networks areused as the access networks between the MR terminals 10 and the IPnetwork is. The access network between the sharing server 20 and the IPnetwork may be either a wired or wireless access network.

(2) MR Terminal

(2-1) Hardware Configuration

FIG. 2 is a block diagram that shows an example of a hardwareconfiguration of each MR terminal 10 shown in FIG. 1. The MR terminal 10is constituted by, for example, a holographic computer or the like thatis capable of 3D display, and has a hardware processor 11A such as a CPU(Central Processing Unit). A program memory 11B, a data memory 12, acommunication interface 13, an input device 14, a display device 15, anda 3D sensor 16 are connected to the hardware processor 11A via a bus 17.

The program memory 11B is a combination of storage media that are, forexample, a nonvolatile memory such as an SSD (Solid State Drive) to andfrom which data can be written and read at any time and a nonvolatilememory such as a ROM (Read Only Memory), and a program required toexecute various kinds of processing is stored therein.

The data memory 12 is a combination of storage media that are, forexample, a nonvolatile memory such as an SSD to and from which data canbe written and read at any time and a volatile memory such as a RAM(Random Access Memory), and is used to store data acquired and createdin the course of various kinds of processing.

The communication interface 13 enables data to be transmitted to andreceived from the sharing server 20 via the communication interface NW.A protocol defined by the communication network NW is used as thecommunication protocol. For example, an interface that employs alow-power wireless data communication standard, such as a wireless LANor Bluetooth (registered trademark), can be used as the communicationinterface 13.

The display device 15 is a user interface that presents a mixed realityspace constructed through processing performed by the processor 11A tothe user. The display device 15 may be a head-mounted display in whichthe mixed reality space is projected onto a transmission holographiclens, for example.

The input device 14 is a user interface for accepting an instructionfrom the user and notifying the processor 11A of the instruction. Theinput device 14 may include a keyboard, a touch panel, an operationbutton, a pointing device such as a mouse, or the like.

The 3D sensor 16 is a sensor for acquiring a real space as 3Dinformation. The processor 11A can perform processing to mix a virtualspace with a real space and construct a mixed reality space by creatinga 3D model of the real space based on 3D information regarding the realspace acquired by the 3D sensor 16 and mapping the virtual informationonto the 3D model.

(2-2) Software Configuration

FIG. 3 is a block diagram that shows a software configuration of the MRterminals 10 shown in FIG. 1 in association with the hardwareconfiguration shown in FIG. 2.

The storage area of the data memory 12 includes a 3D information storageunit 121, an anchor information storage unit 122, and a virtualinformation storage unit 123.

The 3D information storage unit 121 is used to store 3D informationregarding the real space. The 3D information regarding the real spacestored in this 3D information storage unit 121 may be that acquired bythe 3D sensor 16, or that acquired by another MR terminal 10 anddownloaded from the sharing server 20 via the network NW.

The anchor information storage unit 122 is used to store anchorinformation. The anchor information stored in the anchor informationstorage unit 122 may be that generated by this MR terminal 10, or thatgenerated by another MR terminal 10 and downloaded from the sharingserver 20 via the network NW.

The virtual information storage unit 123 stores virtual information thatis mapped onto the 3D model of a real space that is created based on 3Dinformation regarding the real space. In this embodiment, any method ofcreating virtual information may be employed. The virtual informationmay be created by this MR terminal 10, or may be downloaded from anexternal device via the network NW or the like.

The processing unit 11 is constituted by the aforementioned hardwareprocessor 11A and program memory 11B, and includes processing functionalunits realized by software, namely a 3D information acquisition unit111, an initial setting unit 112, a display information generation unit113, a 3D information change detection unit 114, and an updatedinformation upload unit 115. All of these processing functional unitsare realized by causing the aforementioned hardware processor 11A toexecute the program stored in the program memory 11B. The processingunit 11 may also be realized in any of various other forms, includingintegrated circuits such as an ASIC (Application Specific IntegratedCircuit) and an FPGA (field-programmable gate array).

The 3D information acquisition unit 111 performs processing to store 3Dinformation regarding the real space acquired by the 3D sensor 16 in the3D information storage unit 121.

FIG. 4A is a block diagram that shows a configuration of the initialsetting unit 112. This initial setting unit 112 includes a room checkunit 1121, a room creation request unit 1122, an anchor informationupload unit 1123, an anchor information download unit 1124, a 3Dinformation check unit 1125, a 3D information upload unit 1126, and a 3Dinformation download unit 1127.

The room check unit 1121 performs processing to make an inquiry aboutwhether or not a mixed reality space recognized by this MR terminal 10is managed by the sharing server 20, to the sharing server 20 via thecommunication network NW using the communication interface 13. That isto say, the room check unit 1121 performs processing to make an inquiryabout the existence of a room corresponding to the mixed reality spacerecognized by this MR terminal 10 to the sharing server 20. This inquirycan be made by transmitting a check message that includes the name ofthe room to the sharing server 20.

The room creation request unit 1122 performs processing to give aninstruction to create a new room to the sharing server 20 via thecommunication network NW using the communication interface 13 if theroom check unit 1121 receives a reply indicating that no correspondingroom is present from the sharing server 20.

The anchor information upload unit 1123 performs processing to generateanchor information in the mixed reality space recognized by this MRterminal 10, similarly if the room check unit 1121 receives a replyindicating that no corresponding room is present from the sharing server20. The anchor information upload unit 1123 then performs processing toupload the generated anchor information to the sharing server 20 via thecommunication network NW using the communication interface 13. At thistime, the anchor information upload unit 1123 also performs processingto store the generated anchor information in the anchor informationstorage unit 122.

The anchor information download unit 1124 performs processing to make arequest to transmit anchor information, to the sharing server 20 via thecommunication network NW using the communication interface 13 if theroom check unit 1121 receives a reply indicating that the room existsfrom the sharing server 20. The anchor information download unit 1124then performs processing to download the anchor information transmittedfrom the sharing server 20 and store the downloaded anchor informationin the anchor information storage unit 122.

The 3D information check unit 1125 performs processing to make aninquiry about whether or not 3D information relating to the mixedreality space recognized by this MR terminal 10 is managed by thesharing server 20, to the sharing server 20 via the communicationnetwork NW using the communication interface 13.

The 3D information upload unit 1126 performs processing to upload 3Dinformation regarding the real space stored in the 3D informationstorage unit 121 to the sharing server 20 via the communication networkNW if the 3D information check unit 1125 receives, from the sharingserver 20, a reply that is a request to transmit 3D information due tothe absence of the 3D information.

If the 3D information check unit 1125 receives a reply that is 3Dinformation regarding the real space uploaded by another MR terminal 10due to the existence of 3D information from the sharing server 20, the3D information download unit 1127 performs processing to download thecorresponding 3D information from the sharing server 20 via thecommunication network NW using the communication interface 13. The 3Dinformation download unit 1127 performs processing to store the download3D information regarding the real space in the 3D information storageunit 121.

The display information generation unit 113 performs processing togenerate a 3D model of a real space based on 3D information regardingthe real space stored in the 3D information storage unit 121.Furthermore, the display information generation unit 113 performsprocessing to mix a virtual space with a real space to construct a mixedreality space by mapping a virtual object corresponding to virtualinformation stored in the virtual information storage unit 123 to theabove-created 3D model. Here, the display information generation unit113 arrange the virtual object at a relative position from coordinatescalculated based on the anchor information stored in the anchorinformation storage unit 122. The display information generation unit113 then performs processing to present the constructed mixed realityspace to the user using the display device 15.

The 3D information change detection unit 114 performs processing todetect whether or not the 3D information has changed, based on thecurrent 3D information regarding the real space acquired by the 3Dinformation acquisition unit 111 and the 3D information regarding thereal space stored in the 3D information storage unit 121. If it isdetected that the 3D information has changed, the 3D information changedetection unit 114 performs processing to update the 3D informationregarding the real space stored in the 3D information storage unit 121with the current 3D information regarding the real space acquired by the3D information acquisition unit 111 and store the updated 3D informationin the 3D information storage unit 121. Furthermore, if it is detectedthat the 3D information has changed, the 3D information change detectionunit 114 performs processing to supply the current 3D informationregarding the real space to the updated information upload unit 115. Inthis case, however, the 3D information change detection unit 114supplies, to the updated information upload unit 115, only a portion ofthe 3D information that corresponds to the portion at which the changehas been detected, rather than the entire current 3D informationregarding the real space, as will be described in detail later.

The updated information upload unit 115 performs processing to uploadthe information of the changed portion of the current 3D informationregarding the real space supplied from the 3D information changedetection unit 114, to the sharing server 20 via the communicationnetwork NW using the communication interface 13.

Note that the processing unit 11 of each MR terminal 10 of the presentinvention is realized by the processor 11A, which is a computer, and theprogram stored in the program memory 11B. This program can also beprovided to the MR terminal 10 by being recorded in a non-transitorycomputer-readable medium, or via a network. The thus-provided programcan be stored in the program memory 11B. Alternatively, the processor11A can also function as the processing unit 11 as a result of theprovided program being stored in the data memory 12, which is a storage,and executed by the processor 11A as necessary.

(3) Sharing Server

(3-1) Hardware Configuration

FIG. 5 is a block diagram that shows an example of a hardwareconfiguration of the sharing server 20 shown in FIG. 1.

The sharing server 20 is constituted by, for example, a server computer,a personal computer, or the like, and has a hardware processor 21A, suchas a CPU. A program memory 21B, a data memory 22, and a communicationinterface 23 are connected to the hardware processor 21A via a bus 24.

The program memory 21B is a combination of storage media that are, forexample, a nonvolatile memory such as an HDD (Hard Disk Drive) or an SSDto and from which data can be written and read at anytime and anonvolatile memory such as a ROM, and a program required to executevarious kinds of processing is stored therein.

The data memory 22 is a combination of storage media that are, forexample, a nonvolatile memory such as an HDD or an SSD to and from whichdata can be written and read at any time and a volatile memory such as aRAM, and is used to store data acquired and created in the course ofperforming various kinds of processing.

The communication interface 23 enables data to be transmitted to andreceived from the MR terminals 10 via the communication network NW. Aprotocol defined by the communication network NW is used as thecommunication protocol. For example, a wired LAN is used as thecommunication interface 23. An interface that employs a low-powerwireless data communication standard, such as a wireless LAN orBluetooth (registered trademark), may be used as the communicationinterface 23.

(3-2) Software Configuration

FIG. 6 is a block diagram that shows a software configuration of thesharing server 20 shown in FIG. 1 in association with the hardwareconfiguration shown in FIG. 5.

As mentioned above, the sharing server 20 can communicate with theplurality of MR terminals 10#1, 10#2, 10#n via the communication networkNW.

The storage area of the data memory 22 includes a shared informationstorage unit 221. This shared information storage unit 221 is used tostore shared information relating to a room that is created inassociation with 3D information and anchor information transmitted fromone MR terminal 10.

The processing unit 21 is constituted by the aforementioned hardwareprocessor 21A and program memory 21B, and includes processing functionalunits realized by software, namely a room management unit 211, a roomcreation unit 212, an anchor information sharing unit 213, a 3Dinformation management unit 214, a 3D information sharing unit 215, andan updated information management unit 216. All of these processingfunctional units are realized by causing the aforementioned hardwareprocessor 21A to execute the program stored in the program memory 21B.The processing unit 21 may alternatively be realized in any of variousother forms, including integrated circuits such as an ASIC and an FPGA.

If a check message containing the name of a room transmitted from theroom check unit 1121 of any of the MR terminals 10 via the communicationnetwork NW is received, the room management unit 211 performs processingto check whether or not shared information regarding the room is storedin the shared information storage unit 221. The room management unit 211then performs processing to return the check result to the MR terminal10 that has transmitted the message, via the communication network NWusing the communication interface 13. Note that if, at this time, theshared information regarding the room is stored in the sharedinformation storage unit 221, the room management unit 211 associatesthe MR terminal 10 with the room by registering ID information foridentifying the MR terminal 10 that has transmitted the check message,to the shared information regarding the room stored in the sharedinformation storage unit 221. In the following, this processing isexpressed as “letting an MR terminal 10 enter a room”. ID informationregarding each MR terminal 10 can be included in the check message andtransmitted from the MR terminal 10.

If an instruction to create a new room that is transmitted from the roomcreation request unit 1122 of any of the MR terminals 10 via thecommunication network NW is received, the room creation unit 212performs processing to create a new room. That is to say, the roomcreation unit 212 performs processing to reserve an area for storingshared information regarding the new room in the shared informationstorage unit 221.

If anchor information relating to the new room that is transmitted fromthe anchor information upload unit 1123 of any of the MR terminals 10via the communication network NW is received, the anchor informationsharing unit 213 performs processing to store the received anchorinformation in the area for storing shared information regarding the newroom that is reserved in the shared information storage unit 221. Thus,the received anchor information is associated with the new room.Further, if a request to transmit anchor information is received that istransmitted, via the communication network NW, from the anchorinformation download unit 1124 of any of the MR terminals 10 that is ina room regarding which shared information is already stored in theshared information storage unit 221, the anchor information sharing unit213 reads out the anchor information associated with the room from theshared information storage unit 221. The anchor information sharing unit213 then performs processing to transmit the read anchor information tothe MR terminal 10 that has made the request, via the communicationnetwork NW using the communication interface 13.

If an inquiry about whether or not 3D information regarding a real spaceto be shared for a room is managed by the sharing server 20 is received,the inquiry being transmitted, via the communication network NW, fromthe 3D information check unit 1125 of any of the MR terminals 10 thatare in the room, the 3D information management unit 214 performsprocessing to check whether or not the 3D information is stored in theshared information storage unit 221. If the check result indicates thatthe corresponding 3D information is stored, the 3D informationmanagement unit 214 then performs processing to transmit the 3Dinformation to the MR terminal 10 that has made the inquiry, via thecommunication network NW using the communication interface 13.

If 3D information is received that is uploaded via the communicationnetwork NW from the 3D information upload unit 1126 of any of the MRterminals 10 that are in the room, the 3D information sharing unit 215performs processing to store the received 3D information as sharedinformation regarding the corresponding room, in the shared informationstorage unit 221. If the 3D information management unit 214 confirmsthat the 3D information about which the inquiry has been made from theMR terminal 10 is stored in the shared information storage unit 221, the3D information sharing unit 215 performs processing to read out the 3Dinformation from the shared information storage unit 221 and transmitthe read 3D information to the MR terminal 10 that has made the inquiry,via the communication network NW using the communication interface 13.

The updated information management unit 216 receives information of achanged portion of the current 3D information regarding a real spacethat is uploaded from the updated information upload unit 115 of any ofthe MR terminals 10 that are in the room via the communication networkNW. The updated information management unit 216 then performs processingto update, with the received information, the portion of the 3Dinformation regarding the real space stored as shared informationregarding the corresponding room in the shared information storage unit221.

Note that the processing unit 21 of the sharing server 20 of the presentinvention is realized by the processor 21A, which is a computer, and aprogram stored in the program memory 21B, but this program can also beprovided to the sharing server 20 by being recording in a non-transitorycomputer-readable medium, or via a network. The thus-provided programcan be stored in the program memory 21B. Alternatively, the processor21A can also function as the processing unit 21 as a result of theprovided program being stored in the data memory 22, which is a storage,and executed by the processor 21A as necessary.

(Operation)

Next, a description will be given of an operation of the mixed realityspace sharing system that is configured as described above. FIGS. 7A to7C are diagrams that show a series of flowcharts illustrating an exampleoperation in processing performed by the processor 11A (processing unit11) of each MR terminal 10 and a series of flowcharts illustrating anexample operation in processing performed by the processor 21A(processing unit 21) of the sharing server 20. These flowcharts onlyshow portions of processing related to sharing a mixed reality spacebetween a plurality of MR terminals.

It is assumed here that the MR terminal 10 is already in a state ofbeing connected to the sharing server 20 via the communication networkNW through a predetermined procedure. That is to say, the processor 11Aof the MR terminal 10 connected to the sharing server 20 first causesthe room check unit 1121 of the initial setting unit 112 to make aninquiry to the sharing server 20 about whether or not a mixed realityspace recognized by this MR terminal 10 is managed by the sharing server20, that is, about the existence of a room corresponding to the mixedreality space recognized by this MR terminal 10. This inquiry is made bytransmitting a check message containing the name of the room to thesharing server 20 (step S101). Thereafter, the processor 11A causes theroom check unit 1121 to wait for a reply from the sharing server 20(step S102).

The processor 21A of the sharing server 20 waits for reception from theMR terminal 10 in the later-described processing loops in steps S201,S211, and S217.

If the room management unit 211 receives a check message containing thename of the room from the room check unit 1121 of any of the MRterminals 10 (YES in step S201), the processor 21A of the sharing server20 checks whether or not shared information regarding the correspondingroom is stored in the shared information storage unit 221 of the datamemory 22 (step S202). At this time, if the shared information regardingthe corresponding room is stored in the shared information storage unit221, the processor 21A causes the room management unit 211 to associatethe MR terminal 10 that has transmitted the message with the room. Thatis to say, the processor 21A lets the MR terminal 10 that hastransmitted the message enter the corresponding room. The processor 21Athen causes the room management unit 211 to return the check result tothe MR terminal 10 that has transmitted the message (step S203).

Thereafter, the processor 21A determines whether or not the check resultin the above step S202 indicates that the corresponding room exists(step S204). Here, if the check result indicates that the correspondingroom exists (YES in step S204), the processor 21A advances theprocessing to step S209, which will be described later. On the otherhand, if the check result in the above step S202 indicates that thecorresponding room does not exist (NO in step S204), the processor 21Acauses the room creation unit 212 to wait for receiving a room creationrequest from the MR terminal 10 that has transmitted the aforementionedmessage (step S205).

If the room check unit 1121 receives a reply from the sharing server 20(YES in step S102), the processor 11A of the MR terminal 10 that hastransmitted the message determines whether or not the content of thereply indicates that the corresponding room exists (step S103). If thereply indicates that the corresponding room exists (YES in step S103),the processor 11A advances the processing to step S107, which will bedescribed later. On the other hand, if the received reply indicates thatthe corresponding room does not exist (NO in step S103), the processor11A causes the room creation request unit 1122 of the initial settingunit 112 to transmit a room creation request to the sharing server 20such that a new room with the name is created on the sharing server 20(step S104).

If the room creation unit 212 receives the room creation request fromthe MR terminal 10 (YES in step S205), the processor 21A of the sharingserver 20 causes the room creation unit 212 to create the requested newroom (step S206). That is to say the processor 21A causes the roomcreation unit 212 to reserve an area for storing shared informationregarding the requested new room in the shared information storage unit221. Thereafter, the processor 21A causes the anchor information sharingunit 213 to wait for receiving anchor information relating to the newroom from the MR terminal 10 (step S207).

After transmitting the room creation request to the sharing server 20 inthe above step S104, the processor 11A of the MR terminal 10 that hastransmitted the message causes the anchor information upload unit 1123to generate anchor information in the mixed reality space recognized bythis MR terminal 10 by calculation, and store the generated anchorinformation in the anchor information storage unit 122 of the datamemory 12 (step S105). The processor 11A then causes the anchorinformation upload unit 1123 to upload the generated anchor informationto the sharing server 20 (step S106). Thereafter, the processor 11Aadvances the processing to step S113, which will be described later.

If the anchor information sharing unit 213 receives the anchorinformation relating to the new room (YES in step S207), the processor21A of the sharing server 20 causes the anchor information sharing unit213 to register the received anchor information to the sharedinformation storage unit 221 (step S208). That is to say, the processor21A causes the anchor information sharing unit 213 to associate theanchor information with the new room by storing the received anchorinformation in the area for storing shared information regarding the newroom that is reserved in the shared information storage unit 221.Thereafter, the processor 21A advances the processing to step S214,which will be described later.

Meanwhile, if, in the above step S103, a reply indicating that thecorresponding room exists is received from the sharing server 20 (YES instep S103), the processor 11A of the MR terminal 10 causes the anchorinformation download unit 1124 to make, to the sharing server 20, arequest to transmit anchor information regarding the corresponding room(step S107). The processor 11A then causes the anchor informationdownload unit 1124 to wait for receiving the anchor information from thesharing server 20 (step S108).

If, in the above step S204, the check result indicates that thecorresponding room exists (YES in step S204), the processor 21A of thesharing server 20 causes the anchor information sharing unit 213 to waitfor receiving a request to transmit anchor information from any of theMR terminals 10 that are in the room regarding which shared informationis already stored in the shared information storage unit 221 (stepS209). If a request to transmit anchor information is received from theMR terminal 10 (YES in step S209), the processor 21A causes the anchorinformation sharing unit 213 to read out anchor information associatedwith the room from the shared information storage unit 221 and transmitthe read anchor information to the MR terminal 10 that has made therequest (step S210). Thus, if a request for anchor information isreceived from another MR terminal 10 that are in the same room, theprocessor 21A transmits the anchor information to this MR terminal 10.

If, in the above step S108, the anchor information is received from thesharing server 20 (YES in step S108), the processor 11A of the MRterminal 10 causes the anchor information download unit 1124 to downloadthe anchor information, and attach the downloaded anchor information tothe mixed reality space recognized by this MR terminal 10 by storing thedownloaded anchor information in the anchor information storage unit 122of the data memory 12 (step S109). Thus, the MR terminals 10 that haveentered the same room can acquire the same anchor information, and, as aresult, the same mixed reality space can be viewed.

After the anchor information is shared as described above, the processor11A of the MR terminal 10 checks with the sharing server 20 whether ornot 3D information regarding the real space to be shared has beenuploaded. That is to say, the processor 11A causes the 3D informationcheck unit 1125 to transmit, to the sharing server 20, an inquiry aboutwhether or not 3D information regarding the mixed reality spacerecognized by this MR terminal 10 is managed by the sharing server 20(step S110).

After transmitting the anchor information in the above step S210, theprocessor 21A of the sharing server 20 causes the 3D informationmanagement unit 214 to determine whether or not an inquiry about 3Dinformation regarding the real space to be shared for the room has beenreceived from the 3D information check unit 1125 of any of the MRterminals 10 that are in the room (step S211). If no such inquiry hasbeen received (NO in step S211), the processor 21A advances theprocessing to step S217, which will be described later. On the otherhand, if such an inquiry has been received (YES in step S211), theprocessor 21A causes the 3D information management unit 214 to checkwhether or not the 3D information regarding the real space to be sharedfor the room is stored in the shared information storage unit 221 of thedata memory 22 (step S212). If the corresponding 3D information is notstored in the shared information storage unit 221 (NO in step S212), theprocessor 21A advances the processing to step S214, which will bedescribed later. If the corresponding 3D information is stored in sharedinformation storage unit 221 (YES in step S212), the processor 21Acauses the 3D information management unit 214 to read out thecorresponding 3D information from the shared information storage unit221 and transmit the read 3D information to the MR terminal 10 that hasmade the inquiry (step S213). Thereafter, the processor 21A advances theprocessing to step S217, which will be described later.

After transmitting the inquiry about the 3D information in the abovestep S110, the processor 11A of the MR terminal 10 waits for downloadingof the 3D information from the sharing server 20 (step S111). After the3D information is transmitted from the sharing server 20 (YES in stepS111), the processor 11A causes the 3D information download unit 1127 todownload the 3D information and store the downloaded 3D information inthe 3D information storage unit 121 of the data memory 12 (step S112).Thus, the MR terminal 10 can recognize 3D information regarding the realspace that is sensed and uploaded by another MR terminal 10, withoutsensing the real space. Thereafter, the processor 11A advances theprocessing to step S116, which will be described later.

On the other hand, if the 3D information is not uploaded, or if a newroom is created, the MR terminal 10 needs to sense the target realspace. If, in the above step S212, the corresponding 3D information isnot stored in the shared information storage unit 221 (NO in step S212),the processor 21A of the sharing server 20 causes the 3D informationmanagement unit 214 to transmits a 3D information request for uploadingthe 3D information regarding the target real space to the MR terminal 10that has made the inquiry (step S214). Thereafter, the processor 21Acauses the 3D information sharing unit 215 to wait for receiving the 3Dinformation from the MR terminal 10 (step S215).

After uploading the anchor information in the above step S106, theprocessor 11A of the MR terminal 10 causes the 3D information check unit1125 to determine whether or not the 3D information request has beenreceived from the sharing server 20 (step S113). If the 3D informationrequest has not been received (NO in step S113), the processor 11Aadvances the processing to step S116, which will be described later. Onthe other hand, if the 3D information request has been received (YES instep S113), the processor 11A causes the 3D information check unit 1125to notify the user that there is no 3D information on the sharing server20, using the display device 15 or a speaker (not shown), for example,and have the user acquire 3D information by sensing the target realspace using the MR terminal 10 (step S114). The 3D information acquiredby the 3D sensor 16 is stored in the 3D information storage unit 121 ofthe data memory 12 by the 3D information acquisition unit 111.Thereafter, the processor 11A causes the 3D information upload unit 1126to upload the 3D information stored in the 3D information storage unit121 to sharing server 20 (step S115).

If, in the above step S215, the 3D information is received from the MRterminal 10 (YES in step S215), the processor 21A of the sharing server20 causes the 3D information sharing unit 215 to store the 3Dinformation as shared information regarding the corresponding room inthe shared information storage unit 221 of the data memory 22 (stepS216).

The MR terminals 10 can enter the same room and grasp the complete shapeof the real space to be shared in the above-described manner. That is tosay, as for a real space that has been sensed in the past, the other MRterminals 10 can recognize the shape of this space without sensing thespace again. Accordingly, the processor 11A of each MR terminal 10 cancause the display information generation unit 113 to construct a mixedreality space based on 3D information regarding a real space stored inthe 3D information storage unit 121 and virtual information stored inthe virtual information stored in the virtual information storage unit123, and present the constructed mixed reality space to the user usingthe display device 15.

FIG. 8 is a schematic diagram for illustrating processing for sharing 3Dinformation regarding a real space between two MR terminals 10 (first MRterminal 10#1 and second MR terminal 10#2). FIG. 9 is a diagram thatshows an example of shared information managed by the sharing server 20,which is a shared information management server, when the 3D informationregarding the real space is shared between the two MR terminals 10.

For example, it is assumed that the first MR terminal 10#1, which hasfinished sensing of the shape of the target real space, attempts toenter a room for which shared information regarding the real space isnot yet managed by the sharing server 20. In this case, the processor11A of the first MR terminal 10#1 performs the processing in the abovesteps S101 to S106. Thus, the processor 11A generates anchor informationANC #1 in a mixed reality space CS recognized by this MR terminal 10#1,and uploads the generated anchor information as first anchor informationANC #1 to the sharing server 20. Thereafter, the processor 11A of thefirst MR terminal 10#1 performs the processing in the above steps S113to S115. Thus, the processor 11A can upload first 3D information 3D#1,which is 3D information regarding the target real space, to the sharingserver 20. As a result, an ID (TA #1) of the MR terminal that is in theroom, the anchor information (ANC #1), and the 3D information (3D#1) arestored in association with a room ID (RM #1) indicating this room in theshared information storage unit 221 of the sharing server 20, and aremanaged as shared information in the sharing server 20.

Thereafter, if a new user enters a specific area, such as a targetchamber, and starts the second MR terminal 10#2, the processor 11A ofthe second MR terminal 10#2 registers an ID (TA #2) of the second MRterminal 10#2 to the sharing server 20 by performing the processing inthe above steps S101 to S103, and then advances the processing to theabove step S107. The processor 11A of the second MR terminal 10#2 candownload the anchor information (ANC #1) regarding the room from thesharing server 20 through the processing in the above steps S107 toS109. Furthermore, the processor 11A of the second MR terminal 10#2 candownload the 3D information (3D#1) regarding the room from the sharingserver 20 through the processing in the above steps S110 to S112. Thesame virtual object VO can be viewed on the second MR terminal 10#2 inthe same mixed reality space CS while aligning the coordinate axes withthose of the first MR terminal 10#1 by using the downloaded anchorinformation ANC #1 and 3D information 3D#1.

FIG. 10 is a schematic diagram for illustrating processing for sharing3D information regarding the real space when a third MR terminal alsojoins, and FIG. 11 is a diagram that shows an example of sharedinformation managed by the shared information management server when 3Dinformation regarding the real space is shared between the three MRterminals.

If, thereafter, another new user enters the target chamber and startsthe third MR terminal 10#3, the processor 11A of the third MR terminal#2 also registers an ID (TA #3) of the third MR terminal 10#3 to thesharing server 20 and can download the anchor information (ANC #1) andthe 3D information (3D#1) regarding the room from the sharing server 20,similarly to the above-described second MR terminal 10#2. Accordingly,the same virtual object VO can also be viewed on the third MR terminal10#3 in the same mixed reality space CS while aligning the coordinateaxes with those of the first and second MR terminals 10#1 and 10#2 byusing the downloaded anchor information ANC #1 and 3D information 3D#1.

As described above, the second and third MR terminals 10#2 and 10#3 canrecognize the 3D information regarding the target real space, withoutsensing the real space, while recognizing the same absolute coordinatesand coordinate axes as those of the first MR terminal 10#1, using theanchor information (ANC #1) and the 3D information (3D#1) uploaded bythe first MR terminal 10#1 that has the 3D information regarding thetarget real space. Accordingly, the second MR terminal 10#2 and thethird MR terminal 10#3 are enabled to use the mixed reality space in ashort time.

In general, it is said that users start to feel slow when the processingtime exceeds 1 second, and users find it difficult to keep theirattention when the processing time exceeds 10 seconds, (e.g., see“Response Time Limits: Article by Jakob Nielsen” [Online], [Retrieved onJun. 24, 2019], Internet <URL:https://www.nngroup.com/articles/response-times-3-important-li mits/>).According to the present embodiment, if, for example, the sharing server20 is a laptop PC (memory: 16 GB, CPU: Intel Core i7 2.8 GHz, OS:Windows (registered trademark) 10 64 bit) that is connected to HoloLensserving as the first MR terminal 10#1, through Wi-Fi, and 1.56-MB 3Dinformation (number of points: about 50000) is uploaded from the firstMR terminal 10#1 to the sharing server 20 through HTTP communication, itrequires about 1.5 seconds. In addition, if the same 3D information isdownloaded from the sharing server 20 to HoloLens serving as the secondMR terminal 10#2 through HTTP communication, it requires about 0.5seconds. From these results, if, for example, it requires one minute tosense an entire chamber, in the present embodiment, it is possible tostart sharing a mixed reality space about one minute earlier than whenthe entire chamber is sensed with conventional MR terminals 10.

In such a situation, the shape of a real space frequently changes; e.g.,the positions of a desk, a chair, or the like change, and the positionat which a person stands changes. If 3D information regarding the realspace uploaded to the sharing server 20 is not updated in accordancewith these changes in the shape, 3D information indicating a differentshape from the changed shape is transmitted to the MR terminals 10.Consequently, a problem arises in that, when an MR terminal 10 that hasdownloaded such taste information uses a mixed reality space, virtualinformation will be displayed without regard to the shape of the realspace. That is to say, since the shape of the real space frequentlychanges due to movement of a person or an object, 3D informationregarding the real space needs to be constantly updated to the latestinformation in order to form a mixed reality space without any sense ofdiscomfort. However, if 3D information regarding the entire real spaceis updated, processing takes time at all such occasions. For thisreason, in the present embodiment, 3D information regarding the realspace uploaded to the sharing server 20 is updated as follows.

The processor 11A of the MR terminal 10 causes the 3D informationacquisition unit 111 to reacquires the shape of the real space that hasbeen sensed in the direction in which the user is facing by the 3Dsensor 16, with a period of T seconds, for example (step S116).Furthermore, the processor 11A causes the 3D information changedetection unit 114 to compare the acquired 3D information regarding thereal space with the previously acquired 3D information stored in the 3Dinformation storage unit 121 of the data memory 12, and determinewhether or not the shape has changed (step S117). For example, the MRterminal 10, such as HoloLens, recognize the shape of the real space asa set of points and surfaces (point cloud data). FIG. 12 is a diagramthat shows a box that is a real object RO to be sensed, and 3Dinformation 3DI regarding this box. For example, if the box shown inFIG. 12 is sensed by HoloLens, HoloLens recognizes it as a set of pointsand surfaces. Accordingly, the processor 11A causes the 3D informationchange detection unit 114 to check the numbers of points and surfacesincluded in the 3D information 3DI, compare these numbers with thepreviously acquired numbers, and determine whether or not the shape haschanged based on whether or not the number of points or surfaces haschanged from the previously acquired value by a threshold or more. If itis determined that the shape has not changed (NO in step S117), theprocessor 11A repeats the processing from the above step S116.

On the other hand, if it is determined that the shape has changed (YESin step S117), the processor 11A causes the updated information uploadunit 115 to upload, as updated information, information of the changedportion of the 3D information regarding the real space for which it isdetermined that the shape has changed, to the sharing server 20 (stepS118). Thereafter, the processor 11A repeats the processing from theabove step S116.

Note that the processor 11A may alternatively store the 3D informationregarding the real space acquired at the current time by the 3Dinformation acquisition unit in the 3D information storage unit 121 onlywhen the 3D information change detection unit 114 determines that theshape has changed, rather than constantly. Thus, the processing time ofthe processor 11A can be shortened, and the waste of memory life due toupdating of the memory content of the 3D information storage unit 121can be reduced.

Meanwhile, the processor 21A of the sharing server 20 causes the updatedinformation management unit 216 to determines whether or not updatedinformation of the current 3D information regarding the real spaceuploaded from any of the MR terminals 10 that are in the room has beenreceived (step S217). If it is determined that the updated informationhas not been received (NO in step S217), the processor 21A repeats theprocessing from the above step S201. On the other hand, if it isdetermined that the updated information has been received (YES in stepS217), the processor 21A causes the updated information management unit216 to update the corresponding portion of the 3D information regardingthe real space stored as the shared information regarding thecorresponding room in the shared information storage unit 221 with thereceived information of the changed portion of the current 3Dinformation regarding the real space (step S218). Thereafter, theprocessor 21A repeats the processing from the above step S201.

Here, updated information will be described. FIG. 13 is a schematicdiagram for illustrating uploading of 3D information regarding a realspace. 3D information acquired by an MR terminal 10, which is typifiedby HoloLens, is divided into a plurality of groups (groups a to g inthis example) within the MR terminal 10 as shown in FIG. 13. In thepresent embodiment, a target mixed reality space CS is managed whilebeing divided into a plurality of partial spaces PS in accordance withthose groups, and whether or not the shape has changed is determined foreach of the partial spaces. Here, if a chair, which is a real object RO,moves in a direction indicated by an arrow in FIG. 13, the shapes of apartial space PSb and a partial space PSf change. If the MR terminal 10with a sensing function detects these changes in the spatial shape,pieces 3db and 3Df of the 3D information regarding the partial spacesPSb and PSf and the time of acquisition of these pieces of the 3Dinformation are uploaded as updated information to the sharing server20. Thus, 3D information managed by the sharing server 20 can be updatedevery time the shape of the real space changes.

Here, in order for the MR terminal 10 to detect whether or not the 3Dinformation has changed, it is necessary to compare the 3D informationacquired before the shape changes with the 3D information acquired afterthe shape has changed. One method for comparing the 3D information is tosearch for a combination of points that minimizes the distance betweenpoint clouds of the two pieces of 3D information. This is a method inwhich processing for searching for the closest point to a point ai in apoint cloud A out of points B to be compared is performed for allpoints, and the computational complexity is O(N²). For this reason, whenpieces of the 3D information with a large number of points, such as thatof a real space, are compared, the processing time becomes very long.Consequently, if the shape frequently changes, a problem arises in thatupdating of the 3D information cannot keep up with the changes in theshape.

In the present embodiment, a change in the shape of a real space isdetected only by performing lightweight processing to only check anincrease and decrease in the numbers of points and surfaces included inthe 3D information, rather than comparing points. Specifically, the MRterminal 10 senses the shape of the space at T-seconds intervals, andrecords the recognized numbers of points and surfaces in the 3Dinformation. If the absolute value of the amount of change is athreshold or more, it is determined that the shape of the real space haschanged. Since counting up the points and surfaces in each piece of the3D information takes a computational complexity of O(N) at most, changesin the 3D information can be rapidly detected. This example will bedescribed with reference to FIG. 14. FIG. 14 is a diagram forillustrating an example of 3D information change detection performed bythe 3D information change detection unit in FIG. 3. In FIG. 14, the 3Dinformation 3DI regarding the box that is the real object RO in FIG. 12is divided into a plurality of groups as indicated by dotted lines inFIG. 14. 3D information in each group includes the time of acquisitionthereof and the (x, y, z) coordinates CO of each point. FIG. 14 shows astate where the shape of 3D information in a group Obj_a, which is oneof those groups, has changed. With the change in the shape of the 3Dinformation in the group Obj_a, the number of points corresponding tothis group also decreases from n to n-k, as shown in FIG. 14. In thepresent embodiment, if the amount of decrease in points (k in FIG. 14)is a threshold or more, it is determined that the shape has changed inthis group, and only the 3D information in the group Obj_a is uploadedas updated information to the sharing server 20.

(Effects)

As described above, in one embodiment, the sharing server 20 manages 3Dinformation and anchor information obtained by an MR terminal 10 thatfirst entered a specific area, such as a target chamber, sensing a realspace. When an MR terminal 10 attempts to enter the room, the sharingserver 20 transmits the managed 3D information and anchor information tothis MR terminal 10. Accordingly, the other MR terminal 10 can acquirethe 3D information and the anchor information via the sharing server 20.Accordingly, if the real space has been sensed once, users do not needto sense this space in every corner and can therefore quickly startsharing a mixed reality space. Accordingly, according to one embodiment,the 3D information regarding the target real space can be shared betweena plurality of MR terminals without spending a long processing time.

Further, in one embodiment, when an MR terminal 10 senses a real space,the MR terminal 10 calculates a change from the shape of the real spacebefore being sensed, only using the number of points and surfacesincluded in the 3D information. Accordingly, the MR terminal 10 canrapidly determine whether or not the shape has changed. If it isdetermined that a change has occurred, the MR terminal 10 only uploadsperipheral information to the sharing server 20. Thus, informationregarding the real space on the sharing server 20 can be updated in realtime with small traffic. That is to say, according to one embodiment, 3Dinformation regarding a target real space can be shared between aplurality of MR terminals while keeping the information updated, withoutspending a long processing time.

Other Embodiments

Note that this invention is not limited to the above embodiment.

For example, in the above embodiment, updated information is uploadedfrom any of the MR terminals 10 that are in a room to the sharing server20, thereby making it possible to download the latest 3D informationregarding the real space to an MR terminal 10 that comes in and out fromnow on. When receiving the upload of the updated information, thesharing server 20 may notify MR terminals 10 other than the MR terminal10 that has transmitted the updated information that there is an update,such that these MR terminals 10 can also use the updated 3D information.For example, there are cases where, even when the shape of the realobject RO has not changed, and even if there is a position on the realobject RO that cannot be sensed by a certain MR terminal 10, another MRterminal 10 located at a different position from the aforementioned MRterminal 10 may be capable of sensing the real object RO. If this 3Dinformation is downloaded in advance, the time for newly sensing thereal object RO can be omitted when the terminal moves and informationabout the shape of the real object RO that has not been necessary untilnow is required.

In the above embodiment, all of the MR terminals 10 have the 3D sensor16, but MR terminals 10 other than the first MR terminal 10#1 that firstuploads 3D information may alternatively be inexpensive terminals thatdo not have a 3D sensor. That is to say, since the second and third MRterminals 10#2 and 10#3 can recognize 3D information regarding a realspace without sensing this space, these MR terminals 10#2 and 10#3 canacquire the 3D information regarding the real space even if these MRterminals are terminals that do not have the 3D sensor 16 for sensingthe shape of the real space. Accordingly, a mixed reality space can beused and shared without the 3D sensor 16.

In short, this invention is not limited to the above embodiment, and canbe embodied by modifying constituent elements without departing from thegist thereof at the implementation stage. Various inventions can beformed by appropriately combining a plurality of constituent elementsdisclosed in the above embodiment. For example, some of all constituentelements disclosed in the embodiment may be deleted. Furthermore,constituent elements of different embodiments may be combined asappropriate.

REFERENCE SIGNS LIST

-   10, 10#1, 10#2, 10#n MR terminal-   11, 21 Processing unit-   11A, 21A Processor-   11B, 21B Program memory-   12, 22 Data memory-   13, 23 Communication interface-   14 Input device-   15 Display device-   16 3D sensor-   17, 24 Bus-   20 Sharing server-   111 3D information acquisition unit-   112 Initial setting unit-   113 Display information generation unit-   114 3D information change detection unit-   115 Updated information upload unit-   121 3D information storage unit-   122 Anchor information storage unit-   123 Virtual information storage unit-   211 Room management unit-   212 Room creation unit-   213 Anchor information sharing unit-   214 3D information management unit-   215 3D information sharing unit-   216 Updated information management unit-   221 Shared information storage unit-   1121 Room check unit-   1122 Room creation request unit-   1123 Anchor information upload unit-   1124 Anchor information download unit-   1125 3D information check unit-   1126 3D information upload unit-   1127 3D information download unit-   3D#1, 3DI, 3db, 3Df 3D information-   ANC #1 Anchor information-   CO (x, y, z) coordinates-   CS Mixed reality space-   NW Communication network-   PS, PSb, PSf Partial space-   RO Real object-   VO Virtual object

1. A mixed reality space sharing system comprising: a plurality of mixedreality terminals each of which creates a mixed reality space in whichvirtual information is mapped onto 3D information regarding a real spacebased on the 3D information and anchor information for anchoring thevirtual information, and visualizing the mixed reality space; and ashared information management server to which the plurality of mixedreality terminals are connected, wherein a first terminal, which is atleast one of the plurality of mixed reality terminals, includes: a 3Dsensor for sensing the real space and acquiring the 3D information; andan upload unit for transmitting the 3D information acquired by the 3Dsensor and the anchor information to the shared information managementserver, the shared information management server includes: aninformation management unit for creating a room by associating the 3Dinformation and the anchor information transmitted from the firstterminal with each other, and managing the information; and a sharingunit for transmitting the 3D information and the anchor information to asecond terminal different from the first terminal, of the plurality ofmixed reality terminals, and the second terminal includes a downloadunit for downloading the 3D information and the anchor information thatare transmitted from the shared information management server, andcreates the mixed reality space using the downloaded 3D information andanchor information.
 2. The mixed reality space sharing system accordingto claim 1, wherein the first terminal further includes an updatedinformation upload unit for periodically comparing the 3D informationacquired by the 3D sensor with immediately previous information, andtransmitting, as updated information, 3D information of only a portionwith a significant change to the shared information management server,and the shared information management server further includes an updatedinformation management unit for updating the information managed by theinformation management unit with the updated information transmittedfrom the first terminal.
 3. The mixed reality space sharing systemaccording to claim 1, wherein the second terminal further includes: a 3Dsensor for sensing the real space and acquiring the 3D information; andan updated information upload unit for periodically comparing the 3Dinformation acquired by the 3D sensor with immediately previousinformation, and transmitting, as updated information, 3D information ofonly a portion with a significant change to the shared informationmanagement server, and the shared information management server furtherincludes an updated information management unit for updating theinformation managed by the information management unit with the updatedinformation transmitted from the second terminal.
 4. A sharedinformation management server comprising: an information management unitfor creating a room by associating 3D information regarding a real spaceand anchor information for anchoring virtual information with eachother, and managing the information, the 3D information and the anchorinformation being used by a plurality of mixed reality terminals each ofwhich creates a mixed reality space in which virtual information ismapped onto the 3D information based on the 3D information and theanchor information, and visualize the mixed reality space; and a sharingunit for transmitting the 3D information and the anchor information to amixed reality terminal that has made a request, of the plurality ofmixed reality terminals, in accordance with the request.
 5. The sharedinformation management server according to claim 4, further comprising:an updated information management unit for receiving updated informationof a portion of the 3D information managed by the information managementunit, the updated information being transmitted from one of theplurality of mixed reality terminals when the portion of the 3Dinformation has significantly changed, and updating the informationmanaged by the information management unit with the updated information.6. (canceled)
 7. A mixed reality space sharing method to be used in amixed reality space sharing system including: a plurality of mixedreality terminals each of which creates a mixed reality space in whichvirtual information is mapped onto 3D information regarding a real spacebased on the 3D information and anchor information for anchoring thevirtual information, and visualizing the mixed reality space; and ashared information management server to which the plurality of mixedreality terminals are connected, the method comprising: uploading the 3Dinformation acquired by sensing the real space and the anchorinformation to the shared information management server from a firstterminal, which is at least one of the plurality of mixed realityterminals; creating a room by associating the 3D information and theanchor information uploaded from the first terminal with each other, andmanaging the information, in the shared information management server;transmitting the managed 3D information and anchor information to asecond terminal different from the first terminal, of the plurality ofmixed reality terminals, in accordance with a request from the secondterminal; and downloading the 3D information and the anchor informationthat are transmitted from the shared information management server, andcreating the mixed reality space using the downloaded 3D information andanchor information, in the second terminal.
 8. (canceled)